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1.  STATEMENT  OF  THE  PROBLEM  STUDIED 


Gaining  a  detailed  tactical  picture  of  the  modem  battle-space  is  vital  to  the  success  of  any 
military  operation.  This  picture  is  used  to  direct  the  movement  of  assets  and  material 
over  rugged  terrain  during  day  and  night,  in  uncertain  weather  conditions,  taking  account 
of  possible  enemy  locations  and  activity.  To  provide  a  timely  and  accurate  picture  of  the 
battlespace,  most  modern  command  operations  centers  (COC)  have  access  to  a  multitude 
of  systems  which  provide  information  from  many  different  sources  including  eye  witness 
reports,  aerial  photographs,  sonar,  radar,  synthetic  aperture  radar  (SAR),  multi-spectral 
imaging  (MSI),  hyper  spectral  imaging  (HSI),  foliage  penetration  radar  (FOPEN), 
electro-optic  (EO),  infrared  (IR)  and  moving  target  imaging  (MTI)  data.  These  disparate 
and  sometimes  conflicting  sources  must  be  combined  together  to  present  the  tactical 
view.  However,  the  quantities  of  information  are  sufficiently  large  that  it  is  impossible 
for  an  individual  to  be  able  to  collect  and  comprehend  all  the  infonnation.  Typically, 
each  infonnation  source  is  analyzed  individually  by  a  specially  trained  technician.  Even 
then,  fusing  these  disparate  data  sources  is  difficult.  The  goal  is  to  provide  an  overall 
view  of  the  battle-space  that  is  clear,  concise,  coherent,  complete  and  accurate.  However, 
the  effectiveness  of  such  a  view  is  determined  by  its  usability.  If  the  picture  contained 
all  the  information  that  had  been  collected,  the  commanders  would  be  overloaded  by  the 
quantity  of  information. 

Historically,  most  tactical  decision  makings  were  performed  on  a  sand  table,  i.e.  a  box 
filled  with  sand  shaped  to  replicate  the  battlespace  terrain.  Commanders  moved  around 
small  physical  replicas  of  battlefield  objects  to  visualize  battlefield  situations.  Currently, 
these  same  operations  are  carried  out  using  detailed  paper  maps  and  acetate  overlays. 
These  maps  and  overlays  can  take  several  hours  to  print,  distribute  and  update.  To  speed 
up  the  visualization  process,  the  joint  Maritime  Command  Information  System  (JMCIS), 
a  widely  fielded  military  information  system  including  a  visualization  module  was 
developed.  Although  JMCIS  is  extremely  powerful  and  flexible,  it  has  two  significant 
problems  for  battlefield  visualization:  clutter  and  the  loss  of  three  dimensional 
information.  Visualization  difficulties  can  be  reduced  by  replacing  the  two  dimensional 
plan  view  by  a  three  dimensional  display,  as  is  done  in  the  virtual  environment  for 
battlefield  visualization  called  Dragon  that  is  implemented  on  a  Responsive  Workbench 
at  Naval  Research  Labs.  Essentially,  the  Workbench  is  an  electronic  3D  sand  table. 
Applications  in  which  several  users  collaborate  around  a  workspace  such  as  a  table  are 
excellent  candidates  for  the  workbench;  however,  the  Workbench  imagery  is  only  correct 
for  one  or  two  observers,  whose  3D  viewpoints  are  correctly  tracked.  Other  non-tracked 
users  can  observe  the  scene  as  well,  but  their  perception  is  distorted. 

While  the  Dragon  system  presents  a  leap  forward  as  compared  to  sand  tables,  maps  and 
overlays  and  two  dimensional  visualization  systems,  it  can  be  improved  in  several  ways: 
First  and  foremost,  the  workbench  is  most  useful  in  applications  where  several  users 
collaborate  around  a  workspace,  such  as  a  table.  As  such  it  is  not  suitable  for  the  mobile 
soldier,  or  pilot,  or  other  military  personnel  that  cannot  possibly  carry  a  workbench  and 
are  thus  limited  to  lightweight  Personal  Digital  Assistants  (PDA)  or  Head  Mounted 
Displays  (HMD)  for  3D  display  purposes.  For  example,  mobile  PDA  or  HMD  displays 
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can  provide  useful  infonnation  from  both  a  user's  (first-person)  perspective  or  an  aerial 
perspective:  different  presentation  issues  relate  to  these  two  views  of  the  same 
information. 

Second,  the  workbench  can  be  enhanced  with  augmented  reality  techniques  whereby 
actual  imagery  and  video  from  the  battlespace  can  be  registered  with  computer  generated 
terrain  data  stored  in  geospatial  databases  and  displayed  as  the  virtual  environments.  In  a 
mobile  scenario,  this  can  help  the  mobile  individuals  identify  where  they  are  located,  and 
visualize  where  they  are  heading.  For  instance,  a  soldier  in  a  battlefield  might  use  the 
system  to  find  out  what  lies  beyond  the  large  hill  he  sees  in  front  of  him.  In  addition  to 
providing  valuable  navigation  infonnation  to  the  mobile  military  personnel,  the  captured 
imagery/video  from  individual  soldiers  in  the  field  can  be  used  to  update  the  centralized 
or  decentralized  visualization  databases.  An  important  question  to  answer  in  this  context 
is  the  extent  to  which  mobile  users  need  to  update  their  visualization  databases  based  on 
each  other's  visual  imagery.  Generating  and  updating  visualization  databases  using 
synthetic  data  such  as  maps  and  elevation  data,  or  real  data  such  as  images  and  video,  in 
an  automatic,  yet  accurate  and  fast  way  is  an  important  element  of  our  research  agenda. 

Third,  a  battlefield  necessarily  deals  with  uncertainty,  and  it  is  necessary  to  determine 
ways  to  represent  and  encode  the  confidence  level  that  exists  for  each  piece  of  battlefield 
data.  For  example,  as  the  last  reported  position  of  an  entity  ages,  the  uncertainty  of  where 
the  entity  is  currently  located  grows.  In  a  broader  sense,  uncertainty  can  either  result  from 
sensor  errors  or  processing  algorithms  such  as  image  understanding  and  analysis  and 
target  recognition  algorithms.  An  important  challenge  is  to  present  the  level  of 
uncertainty  associated  with  each  object  in  the  virtual  scene  in  a  visually  intuitive  way 
without  cluttering  or  resulting  in  information  overload.  Uncertainty  can  also  be  used  to 
prioritize  the  information  that  needs  to  be  presented  to  the  users. 

Fourth,  time  must  also  become  a  part  of  battlefield  visualization  system.  This  might  be 
used  to  play  back  the  previous  24  hours  or  to  store  and  review  the  plans  for  the  upcoming 
24  hours.  This  necessitates  the  generation  of  4D  models,  three  detennining  space  and  the 
fourth  dimension  giving  time.  Fast,  accurate  and  automatic  model  generation  based  on 
synthetic  data  such  as  maps  and  elevation  data  as  well  as  real  imagery  is  an  important 
research  topic. 

Fifth,  distributed  computing  is  the  direction  in  which  all  military  systems  are  moving. 
This  includes  remote  person-to-person  collaboration  as  well  as  distributing  the  computing 
across  multiple  platforms. 

Based  on  the  above  requirements,  we  have  investigated  a  distributed,  database  system  for 
battlefield  visualization,  tailored  to  the  needs  of  future  mobile  military  personnel.  This 
database  incorporates  and  presents  uncertainty  associated  with  objects  in  the  virtual 
scene,  is  four  dimensional,  and  is  initially  constructed  by  registering  sensor  imagery  to 
ground  control  points  and  reference  imagery,  with  the  additional  input  of  maps  and 
elevation  data.  The  resulting  4-D  model  provides  the  scene  context  for  interaction, 
interpretation,  and  the  visualization  needs  of  the  users.  The  users  not  only  extract  and 
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visualize  information  from  this  system,  but  also  contribute  to  updating  it.  In  doing  so,  4D 
model  updates  are  placed  into  special  geo-referenced  data  structure  necessary  for  real 
time  visual  navigation  of  very  large  data  collections;  time  is  be  treated  on  the  same 
footing  as  spatial  dimensions  so  that  one  can  navigate  through  time  as  well  as  space  or 
jump  from  time  to  time  for  accurately  geo-located  data. 

In  what  follows,  we  provide  highlights  of  our  research  results  for  this  MURI  project; 
these  include  (a)  automatic  3D  modeling  of  urban  environments  (b)  real  time  sensor  data 
fusion  with  3D  models  and  visualization  (c)  mobile  augmented  battlefield  visualization 
(d)  quantifying  and  visualizing  uncertainty  (d)  decision  making  with  uncertain  image  and 
sensor  data.  In  what  follows  we  elaborate  on  each  one  of  these  in  detail. 

2.  SUMMARY  OF  THE  MOST  IMPORTANT  RESULTS 

2.1  Results  in  3D  Modeling 

2.1.1  Automated  3D  City  Modeling 

Many  current  and  future  military  operations  are  likely  to  take  place  in  cities,  requiring  the 
U.S.  military  to  be  prepared  to  engage  in  operations  in  urban  areas.  As  such,  a  major  key 
to  success  in  such  missions  is  the  ability  to  model  real-world  urban  areas  accurately  and 
effectively,  so  as  to  support  US  military  mission  planning,  operations,  and  training.  This 
requires  urban  terrain  mapping,  interrogation,  and  visualization  capabilities,  together  with 
frequent  update  processing.  In  addition  to  military  applications,  three  dimensional  urban 
models  are  used  in  civilian  applications  such  as  urban  planning,  virtual  reality,  gaming 
and  entertainment  industries,  and  simulation  of  the  propagation  of  radio  waves  for  the 
cell  phone  industry. 

Acquisition  of  3D  city  models  has  traditionally  been  difficult  and  time  consuming.  As 
such,  large  scale  models  typically  take  months  to  create,  and  usually  require  significant 
manual  intervention  [ChaShe98].  This  process  is  not  only  prohibitively  expensive,  but 
also  is  unsuitable  in  applications  where  a  3D  snapshot  of  a  city  is  needed  within  a  short 
time,  e.g.  for  disaster  management  or  for  monitoring  changes  over  time. 

There  exist  a  variety  of  approaches  to  creating  3D  models  of  cities  from  an  airborne  view 
via  remote  sensing.  One  approach  is  to  use  aerial  images  in  stereo  vision  algorithms 
[Frere98],  In  recent  years,  advances  in  resolution  and  accuracy  of  airborne  laser  scanners 
have  also  rendered  them  suitable  for  the  generation  of  3D  models 
[Brenner2001,Maas2001].  Although  these  methods  can  be  reasonably  fast,  the  resulting 
resolution  of  the  models  is  not  high,  and  without  manual  intervention,  the  resulting 
accuracy  is  also  poor.  Specifically,  they  lack  the  level  of  detail  that  is  required  for 
realistic  virtual  walk-throughs  or  drive-throughs  needed  in  military  applications. 

Previous  work  on  acquiring  detailed  building  models  from  a  ground-level  view  has  been 
limited  to  one  or  few  buildings:  Debevec  et  al.  proposes  to  reconstruct  a  building  based 
on  few  camera  images  in  a  semi-automated  way  [debevec].  Stamos  and  Allen  use  a  3D 
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laser  scanner  [stamos],  and  Thrun  et  al.  [thrun]  and  Hahnel  et  al.  [hahnel]  use  2D  laser 
scanners  mounted  on  a  mobile  robot  to  achieve  complete  automation,  but  the  time 
required  for  data  acquisition  of  an  entire  city  is  prohibitively  large;  Antone  and  Teller 
[antone]  propose  an  approach  based  on  high-resolution  half-spherical  images,  but  data 
has  to  be  acquired  in  a  stop-and-go  fashion. 

Over  the  past  6  years,  under  this  MURI  program,  the  Berkeley  team  has  developed  data 
acquisition  and  processing  techniques  for  fast,  automated,  photorealistic,  3D  modeling  of 
urban  terrain  that  can  be  used  for  virtual  walk-throughs,  drive-throughs,  and  fly-throughs 
[AZ 1  ,AZ2,AZ3,AZ4,AZ5,AZ6,AZ7,AZ8,AZ9,AZ  1 0,AZ  1 1  ,AZ  12,AZ  13].  Our  approach 
uses  laser  and  camera  data  both  at  the  ground  and  aerial  level,  and  consists  of  several 
steps:  (a)  accurate  localization  of  the  ground  acquisition  vehicle;  (b)  processing  of  the 
ground  based  laser  and  camera  imagery  in  order  to  arrive  at  accurate  3D  models  of  the 
facades  of  buildings;  (c)  acquisition  and  processing  of  airborne  laser  and  camera  imagery 
in  order  to  arrive  at  accurate  3D  models  of  rooftops;  (d)  registration  of  the  data  from 
ground  and  aerial  data  with  respect  to  each  other;  (e)  merging  of  the  ground  and  aerial 
models  in  order  to  arrive  at  a  complete  model  of  an  urban  environments;  the  flow 
diagram  for  our  overall  modeling  approach  is  shown  in  Figure  1. 
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Figure  1:  Data  flow  diagram  of  our  modeling  approach.  Airborne  modeling  steps  are 
highlighted  in  green,  ground-based  modeling  steps  in  yellow,  and  model  fusion  steps  in 
white. 
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Our  ground  based  data  acquisition  system  consisted  of  two  fast,  inexpensive  2D  laser 
scanners,  one  horizontal,  and  one  vertical,  and  a  digital  camera.  This  data  acquisition 
system  is  mounted  on  a  truck  moving  at  a  normal  speed  on  public  roads,  collecting  data 
to  be  processed  offline.  This  approach  has  the  advantage  that  data  can  be  acquired 
continuously,  rather  than  in  a  stop-and-go  fashion,  and  is  therefore  much  faster  than 
existing  methods  based  on  3D  scanners.  The  basic  idea  is  to  stack  the  vertical  laser  scans 
acquired  over  time  next  to  each  other  at  proper  positions  in  order  to  arrive  at  a  faithful 
reconstruction  of  the  building  facades.  As  such,  it  is  necessary  to  determine  the  pose  of 
successive  laser  scans  and  camera  images  in  a  global  coordinate  system  with  centimeter 
and  sub-degree  accuracy.  We  have  accomplished  this  by  computing  relative  position 
changes  via  matching  successive  horizontal  laser  scans  against  each  other  [AZ2]. 
Furthermore,  we  have  developed  a  novel  particle  filtering  based  method  to  correct 
accumulating  pose  uncertainty  by  using  airborne  data  such  as  aerial  photo  or  a  digital 
surface  model  (DSM)  as  a  reference  [AZ13].  An  inherent  advantage  of  this  approach  is 
that  it  automatically  registers  ground  level  laser  scans  and  camera  images  with  airborne 
data,  facilitating  subsequent  fusion  of  the  aerial  and  ground  models  in  later  steps  [AZ3]. 

Once  the  ground  based  vehicle  is  accurately  localized,  it  is  possible  to  stack  successive 
vertical  laser  scans  to  construct  a  structured  point  cloud  of  the  facades  of  the  buildings. 
As  there  are  many  erroneous  scan  points,  e.g.  due  to  glass  surfaces  and  foreground 
objects  partially  occluding  the  desired  buildings,  the  generation  of  a  facade  mesh  is  not 
straightforward.  Specifically,  a  simple  triangulation  of  the  raw  scan  points  by  connecting 
neighboring  points  whose  distance  is  below  a  threshold  value  does  not  result  in  an 
acceptable  reconstruction  of  the  street  scenery.  In  [AZ1],  we  have  developed  a  class  of 
data  processing  algorithms  to  create  visually  appealing  facade  models  from  the  ground 
based  data  by  first  converting  our  triangulated  3D  model  into  a  2.5  dimensional  depth 
map,  and  introducing  a  representation  based  on  multiple  depth  layers  of  the  street 
scenery.  Each  depth  layer  is  a  scan  grid,  and  the  scan  points  of  the  original  grid  are 
assigned  to  exactly  one  of  the  layers.  If  at  a  certain  grid  location  there  is  a  point  in  a 
foreground  layer,  this  location  is  empty  in  all  layers  behind  it,  and  needs  to  be  filled  in.  In 
practice,  we  have  found  it  to  be  sufficient  to  generate  only  two  layers,  namely  a 
foreground  and  a  background  layer.  We  then  apply  depth  based  histogram  analysis  to 
separate  foreground  objects  such  as  trees,  cars,  pedestrians,  telephone  poses,  from  those 
of  the  background,  i.e.  building  facades.  Subsequently,  erroneous  scan  points  in  the 
background  layer  are  detected  and  removed,  the  occluding  foreground  layer  is  segmented 
into  objects,  and  occlusion  holes  resulting  from  such  objects  are  filled  using  a  RANSAC 
based  plane  fitting  algorithm  [AZ1]. 

In  [AZ3],  we  develop  a  series  of  processing  steps  for  airborne  laser  data  consisting  of  (a) 
re-sampling  and  interpolation  of  the  raw  laser  data;  (b)  segmentation  of  the  rooftops 
based  on  depth  discontinuity;  (c)  removing  small  segments  such  as  ventilation  ducts;  (d) 
hole  filling  by  extrapolation  of  nearby  planar  segments;  (e)  finding  polygonal 
approximation  for  segment  perimeter  based  on  RANSAC,  and  (f)  straightening  of  the 
edges;  Figures  2(a)  and  2(b)  show  the  triangulation  of  the  airborne  Berkeley  data  and  its 
postprocessed  version  respectively.  As  seen,  the  above  processing  steps  result  in  a 
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dramatic  improvement  in  the  visual  quality  of  the  model.  Figure  2(c)  shows  the  texture 
mapped  version  of  the  processed  DSM  via  airborne  top  down  imagery. 


(c) 


Figure  2:  Airborne  model,  (a)  DSM  directly  triangulated,  (b)  triangulated  after 
postprocessing,  (c)  model  texture-mapped. 

Once  the  airborne  and  ground  based  models  have  been  generated,  they  need  to  be 
combined  to  generate  a  complete  model.  Since  the  two  models  have  been  registered  with 
respect  to  each  other  in  the  ground  vehicle  localization  step  described  earlier,  no  further 
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registration  is  needed.  However,  merging  of  the  two  models  is  still  a  particularly 
challenging  task  since  the  resolutions  of  the  two  models  are  vastly  different,  with  the 
resolution  of  the  facade  model  being  about  10  centimeters,  and  that  of  the  airborne  model 
around  half  a  meter.  To  enable  interactive  rendering,  the  two  models  need  to  fit  together 
even  if  their  parts  are  at  different  levels  of  detail  or  resolutions.  To  address  this  problem, 
we  have  developed  a  “blend  mesh”  approach  to  fill  in  the  gaps  that  inevitably  exist 
between  the  two  models  [AZ3].  An  example  of  walk-through  and  fly  through  views  of 
the  model  for  a  portion  of  University  Avenue  in  downtown  Berkeley  as  seen  from  the 
ground  level  is  shown  in  Figure  3.  As  seen,  the  model  is  quite  photo-realistic,  and  can  be 
used  for  virtual  navigation  through  streets  of  Berkeley  in  walk/drive/fly-through  modes. 


Figure  3:  Examples  of  Walkthrough  and  Fly-through  for  downtown  Berkeley 
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Resulting  3D  models  of  downtown  Berkeley  can  be  downloaded  from: 
http://www-video.eecs.berkeley.edu/~frueh/3d/index.html 

In  addition,  a  Quicktime  video  of  interactive  walk/drive/fly  through  for  downtown 
Berkeley  can  be  downloaded  from: 

http://www-video.eecs.berkeley.edu/~avz/downl  1  .mov 

In  the  past  two  years,  we  have  further  extended  our  basic  modeling  approach  in  several 
directions: 

2.1.2  Extension  to  suburban  and  residential  areas  with  trees 

We  have  developed  an  approach  to  detecting  trees  in  registered  aerial  image  and  range 
data  obtained  via  LiDAR  [AZ12,AZ13].  Representing  the  trees  in  3D  models  is 
problematic  because  the  data  are  usually  too  sparsely  sampled  in  tree  regions  to  create  an 
accurate  3-D  model  of  the  trees.  Furthermore,  including  the  tree  data  points  interferes 
with  the  polygonization  step  of  the  building  roof  top  models.  Therefore,  it  is 
advantageous  to  detect  and  remove  points  that  represent  trees  in  both  LiDAR  and  aerial 
imagery.  We  have  developed  a  two-step  method  for  tree  detection  consisting  of 
segmentation  followed  by  classification.  The  segmentation  is  done  using  a  simple  region 
growing  algorithm  using  weighted  features  from  aerial  image  and  LiDAR,  such  as  height, 
texture  map,  height  variation,  and  normal  vector  estimates.  The  weights  for  the  features 
are  determined  using  a  learning  method  on  random  walks.  The  classification  is  done 
using  weighted  support  vector  machines  (SVM),  allowing  us  to  control  the 
misclassification  rate.  The  overall  problem  is  formulated  as  a  binary  detection  problem, 
and  the  results  presented  as  receiver  operating  characteristic  curves  are  shown  to  validate 
our  approach.  As  shown  in  Figure  4,  this  method  is  capable  of  segmenting  and  classifying 
trees  in  urban  environments. 
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Figure  4:  Example  of  segmentation  ofUC  Berkeley  campus  data;  green  denotes  correctly 
classified  trees;  pink  shows  non-trees  incorrectly  classified  as  trees;  blue  denotes  trees 
incorrectly  classified  as  non-trees. 


2.1.3  Semi-Automatic  Modeling  of  Cities 

The  USC  team  has  developed  a  complete  modeling  system  that  can  extract  and  refine 
complex  building  structures  with  irregular  shapes  and  surfaces  in  a  semi-automated  way. 
The  global  building  footprints  and  roof  data  provided  by  the  LiDAR  reconstruction  is 
used  to  detennine  the  geo-locations  of  buildings  and  isolate  them  from  surrounding 
terrain.  Based  on  the  shape  of  a  building  roof  (flat-roof,  slope-roof,  dome-roof,  gable- 
roof,  etc.),  we  divide  a  complex  building  into  several  basic  building  primitives  (including 
the  standard  CG  primitives  such  as  plane,  cube,  wedge,  polyhedron,  cylinder  and  sphere, 
and  high-order  surface  primitives  such  as  superquadrics)  and  model  them  using  a 
parametric  representation.  As  the  models  from  constructive  solid  geometry  allow  the 
composition  of  complex  models  from  basic  primitives  that  are  represented  as  parametric 
models,  our  approach  is  quite  general.  Also,  as  the  type  of  primitive  is  not  limited,  may 
contain  objects  with  curved  surfaces,  so  the  flexibility  of  model  combinations  is  very 
high,  hence  we  can  model  a  range  of  complex  buildings  with  irregular  shapes  and 
surfaces  by  combining  appropriate  geometry  primitives  and  fitting  strategies. 

This  system  is  semi-automatic  in  that  it  requires  user  interaction  to  select  the  building 
section  and  associated  group  type.  Once  the  user  input  is  provided,  the  system 
automatically  performs  the  processing  including  boundary  segmentation,  primitive 
fitting,  model  refinement,  and  optimization  to  complete  the  building  model.  Our  system 


9 


also  provides  a  range  of  editing  tools,  allowing  users  to  further  refine  the  models  or 
obtain  a  specific  representation  quickly.  The  system  has  been  tested  using  a  range  of 
different  building  structures.  Figure  shows  the  result  of  modeling  entire  USC  campus 
and  surrounding  University  Park  area  including  the  Coliseum,  LA  Arena,  museums,  and 
gardens. 


Figure  5  :  Complete  refined  models  of  USC  campus  and  surrounding  University  Park 
area  (left).  An  aerial  photograph  is  projected  on  the  models  heightening  the  realism  of 
models  (right).  The  model  was  created  in  two  days  using  the  developed  system 

2.2  Results  on  4D  capture  and  Modeling 
2.2.1  Augmented  Virtual  Environment  (AVE) 

Dynamic  scene  analysis  and  object  extraction  are  traditional  problems  in  computer 
vision.  We  have  introduced  the  concept  of  Augmented  Virtual  Environment  (AVE)  as  the 
framework  for  incorporating  our  proposed  model/image/video/data  fusion  techniques  and 
algorithms.  The  AVE  is  a  novel  and  comprehensive  approach  to  data  fusion,  analysis,  and 
visualization  that  incorporates  and  presents  all  the  sensors,  abstract  data,  objects,  and 
scenes  models  within  a  common  context  to  produce  a  concise,  coherent,  and  non- 
conflicting  representation  for  time-space  interpretation  of  real  world  activity.  The  AVE 
framework  is  particularly  suited  to  addressing  the  difficult  problem  posed  by  multiple 
video  sources.  In  an  AVE  system,  image  sensors  are  modeled  as  “virtual  projectors” 
(VPs)  that  have  the  same  imaging  parameters  as  the  sensors. 

The  overall  approach  is  the  fusion  of  video  streams  in  the  context  of  3D  models.  In  order 
to  achieve  that,  several  problems  had  to  be  addressed  including:  rapid  3D  modeling  of  the 
site  in  order  to  provide  the  context  for  video  fusion;  efficient  encoding/decoding  and 
transfer  of  video  into  the  host  computer;  real-time  texture  projection  onto  3D  models;  and 
texture  management  for  scalable  performance  to  100’s  of  cameras.  Sub-barriers  within 
the  problem  of  rapid  and  accurate  modeling  of  terrain  and  structures  include  data  fusion 
and  integration  of  multiple  data  sources  (images  and  LiDAR)  as  well  as  integrated 
automated  tools  for  rapid  processing  of  data. 
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Our  research  activities  resulted  in  the  development  of  methods  and  algorithms  for 
scalable  video  acquisition  and  texture  management.  These  new  algorithms  are  now  a  part 
of  the  AVE  system,  automatically  managing  the  compression  quality,  video  frame  rate, 
and  video  image  size  of  each  video  stream  so  that  a  workload  ceiling  on  the  AVE  system 
is  never  exceeded.  A  limit  on  the  system  workload  means  that  real-time  rendering  and 
interaction  with  the  user  is  maintained  for  an  arbitrarily  high  number  of  camera  video 
streams. 

Encoding/decoding  of  video  Initial  efforts  to  move  video  into  a  computer  and  apply  it  as 
a  dynamic  texture  proved  to  be  cumbersome  and  slow.  The  movement  of  large  volumes 
of  data  exceeded  the  limits  of  computer  systems.  By  careful  testing,  and  engineering,  we 
leveraged  the  available  high  performance  subsystems  in  current  computers  and  graphics 
cards,  as  well  as  the  emerging  commercial  trend  for  video-over-IP  products. 
Multithreading  and  multiple  CPU  or  CPU  Core  systems  allow  for  efficient  utilization  and 
parallel  decoding  of  video.  High  bandwidth  networks  (Gigabit  Ethernet)  provide 
commercial  solutions  to  video  routing.  Optimized  JPEG  and  MPEG  software  decoders 
provide  high  performance  decoding  of  multiple  video  streams. 

Real-time  texture  projection  onto  3D  models  Once  decoded  video  is  available  in  a  host 
computer,  its  projection  onto  a  3D  model  requires  high  bandwidth  data  transfer  and  high 
throughput  graphics  rendering  with  textures.  We  optimize  our  algorithms  to  make  the 
most  of  the  high-performance  AGP  and  PCIE  bandwidth  to  graphics  cards  and  their  high 
rendering  performance.  We  developed  visibility  culling  and  multiscale  texturing 
algorithms  to  maintain  high  throughput  under  all  viewing  conditions.  We  developed 
parallel  shader  GPU  programs  to  accelerate  the  texture  projection  processing. 

Texture  management  for  scalable  performance  to  100’s  of  cameras  We  developed 
methods  and  algorithms  for  scalable  video  acquisition  and  texture  management.  These 
algorithms  automatically  manage  the  compression  quality,  video  frame  rate,  and  video 
image  size  of  each  video  stream  so  that  a  workload  ceiling  on  the  AVE  system  is  never 
exceeded.  A  limit  on  the  system  workload  means  that  real-time  rendering  and  interaction 
with  the  user  is  maintained  for  an  arbitrarily  high  number  of  camera  video  streams. 

2.2.2  Dynamic  Scene  Capturing  System 

The  UC  Berkeley  team  has  started  to  investigate  a  new  approach  for  capturing  dynamic 
scenes  such  as  a  basketball  game  or  soldiers  fighting.  Our  goal  is  to  acquire  both  3D 
geometry  and  visual  appearance  of  a  scene  over  time,  and  thus  essentially  record  a  4D 
movie  which  could  be  viewed  afterwards  from  any  arbitrary  position.  Our  approach 
utilizes  a  video  camera,  an  infrared  (IR)  camera,  a  pattern  of  vertical  IR  light  stripes,  and 
a  2D  IR  line  laser  with  a  rotating  mirror  in  a  setup  as  shown  in  Figure  6Figure( a).  We 
use  structured  IR  light  in  order  to  avoid  interfering  with  or  disturbing  the  moving  persons 
or  objects. 

An  invisible  static  line  pattern  of  equally  spaced  vertical  lines  is  projected  onto  the  scene 
with  the  IR  line  projector,  and  this  line  pattern  is  recorded  from  a  different  position  at  15 
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Hz  frame  rate  with  the  infrared  camera.  Due  to  the  different  viewpoint,  the  projected  lines 
are  curved  in  the  images  according  to  the  shape  of  the  objects,  and  we  track  each  of  them 
and  obtain  a  set  of  curved  lines.  If  we  manage  to  identify  the  IR  light  plane  equation  for 
each  curved  line  in  the  IR  image,  we  can  reconstruct  the  depth  of  the  objects  along  the 
lines  based  on  triangulation.  However,  identifying  the  corresponding  IR  light  plane  is  a 
non-trivial  problem;  in  previous  work,  solutions  such  as  line  coding  or  time  multiplexing 
have  been  proposed,  which  are  not  applicable  for  IR  systems  and  dynamic  scenes, 
respectively.  In  our  approach,  we  attempt  to  solve  this  identification  problem  via  a 
horizontal  IR  line  laser. 

Using  a  rotating  mirror,  we  sweep  the  IR  laser  line  vertically  over  the  scene  at  a  low 
frame  rate,  e.g.  1Hz.  Since  this  is  the  only  horizontal  IR  line  in  the  image,  it  is  easy  to 
identify.  Knowing  the  angle  of  the  rotating  mirror,  we  can  compute  the  depth  accurately 
along  this  line.  For  the  vertical  lines  which  intersect  with  this  laser  line,  we  can  compute 
the  IR  light  plane  equation  by  using  the  3D  coordinates  of  the  intersection  point,  the 
center-of-projection  of  the  IR  light  projector,  and  the  verticality  of  the  light  planes  (intra¬ 
frame  line  tracking).  For  vertical  lines  that  do  not  intersect  with  the  horizontal  line,  we 
determine  vertical  lines  in  the  previous  frame  which  were  close,  and  utilize  their  plane 
equation;  this  is  possible  because  the  motion  between  2  frames,  i.e.  within  66  ms,  is 
limited.  Since  the  horizontal  line  sweeps  continuously  across  the  scene,  the  plane 
equation  for  every  line  segment  is  identified  at  some  point;  by  tracking  lines  across  the 
frames  (inter-frame  line  tracking),  we  can  pass  on  the  plane  equation  from  one  frame  to 
the  next  one. 

Our  results  on  this  work  are  preliminary  but  promising.  Figure  6(b)  shows  a  frame  from 
the  IR  video  of  our  acquisition  system,  and  Figure  6(c)  the  reconstructed  depth  of  the 
moving  object. 


Figure  6:  Dynamic  scene  capturing:  (a)  setup,  (b)  IR  camera  view  with  projected  vertical 
stripes  and  the  horizontal  laser  line,  (c)  reconstructed  depth. 


A  video  of  interactive  rendering  of  a  time  varying  human  object  in  a  room  can  be 
downloaded  from: 

http  ://www-video .  eecs.berkelev.edu/~avz/NewDynSceneMovie.avi 
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2.3.  Results  on  Mobile  Situational  Visualization 


Accomplishments  of  this  MURI  program  on  mobile  situational  visualization  include: 

•  Development  of  a  testbed  for  mobile  augmented  battlefield  visualization. 

•  Creation,  development,  and  evaluation  of  “mobile  situational  visualization”. 

o  Development  of  a  mobile  prototype  including  a  command  center,  server 
architecture,  and  connected,  collaborating  mobile  participants  with  GPS, 
orientation  tracking,  and  shared  annotations, 
o  Testing  and  evaluation  of  the  mobile  environment  with  tracking  scenarios  to 
show  capabilities  and  effectiveness. 

•  In  conjunction  with  Syracuse  University,  added  dynamic  decision  support  to  mobile 
situational  visualization  so  that  paths  could  be  chosen  under  the  placement  of  moving, 
changing  risks.  Paths  were  chosen  in  a  trade-off  of  lowered  risk  versus  shorter 
distance. 

•  Developed  new  simplification  methods  for  detailed  3D  building  models  so  that  they 
could  be  rendered  quickly  with  appropriate  appearance  characteristics. 

o  Punctuated  simplification  method  for  producing  significant,  even  extreme, 
simplifications  of  manmade  models  while  preserving  shape, 
o  Appearance-preserving  level  of  detail  method  for  producing  continuous  LOD 
simplifications  that  applies  an  error  metric  to  both  geometry  and  appearance 
(e.g.,  textures). 

•  Created  a  novel  approach  that  uses  the  principals  of  urban  legibility  to  detennine 
levels  of  abstraction  for  completely  free  navigation  of  massive  urban  models  (with 
hundreds  of  thousands  of  buildings).  With  this  approach  skylines  and  neighborhoods 
are  simplified  in  such  a  way  that  the  overall  structure  of  the  city  remains  legible  and 
understandable  while  close-up  details  are  retained. 

•  Developed  methods  to  automatically  generate  dense  urban  models.  A  variety  of 
sources  including  insurance  databases,  LIDAR  scans,  topographic  data,  and  existing 
models  are  combined  automatically.  The  resulting  urban  model  has  both  generic  and 
specific  structures;  the  former  have  accurate  footprints,  orientation,  locations,  and 
heights.  Generic  roof  and  facade  textures  are  automatically  generated.  Using  these 
methods,  we  generated  extensive  urban  models  for  both  Atlanta  and  Charlotte.  The 
latter  model  has  in  excess  of  370,000  buildings. 

•  Development  of  new  methods  to  automatically  extract  tree  and  shrub  footprints  from 
ortho-rectified  color  aerial  images.  Tree  and  shrub  models  can  then  be  generated  from 
these  with  heights  generate  either  procedurally  from  an  analysis  of  the  tree  width  or 
directly  from  LIDAR  data. 

•  Development  of  new  multimodal  interfaces  for  mobile  situational  visualization. 
Interfaces  include  gesture  tracking  using  a  chest  pendant  with  camera,  hand 
orientation  tracking,  and  tablet-based  sketch  input. 

•  Presented  mobile  situational  visualization  and  its  relation  to  homeland  security  at 
invited  talks  at  a  special  session  of  the  AAAS  meeting  on  the  National  Visualization 
and  Analytics  Center  (February,  2005),  at  AppliedVis  2005  (May,  2005),  and  at  an 
invited  presentation  for  the  DHS  Regional  Visualization  and  Analytics  Centers 
(January,  2006). 
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•  Using  work  in  urban  terrain  analysis  begun  under  this  MURI  as  a  foundation,  began 
work  on  a  project  sponsored  by  ARO  for  eye-point  dependent  models  applied  to 
terrain  analysis  and  applications  such  as  line-of-sight. 

•  Based  on  discussions  with  the  program  manager  at  DARPA  IXO,  we  were 
encouraged  to  submit  a  proposal  on  mobile  situational  visualization  (in  conjunction 
with  Syracuse  University).  Although  the  proposal  was  not  funded,  we  are  continuing 
to  explore  these  possibilities  with  DARPA. 

•  We  presented  our  mobile  system  and  visualization  capabilities  to  many  individuals  in 
state  and  national  government  and  also  to  the  military.  The  system  was  also  a  key 
presentation  during  the  Georgia  Tech  Homeland  Defense  Workshop.  The  system  is 
being  used  as  part  of  the  Samoff  Raptor  system,  which  is  deployed  to  the  Army  and 
other  military  entities.  In  addition  our  visualization  system  is  being  used  as  part  of  the 
Raptor  system  at  Scott  Air  Force  Base. 

•  VGIS,  which  is  the  foundation  for  our  mobile  visualization  system,  was  licensed  by 
Sarnoff  Corp.  It  is  being  used  in  a  system  that  provides  an  integrated  picture  of 
aircraft  operations  and  ground  movements. 

•  Our  system  was  presented  as  part  of  a  mobile  emergency  response  application  to 
President  Bush  and  Governor  Tom  Ridge,  at  that  time  head  of  DHS.  The  presentation 
was  part  of  an  extensive  exercise  for  emergency  response  in  an  urban  setting  to  a 
terrorist  attack,  in  this  case  the  release  of  sarin  gas  inside  and  outside  a  large  building 
in  the  city. 

•  Situational  visualization  and  mobile  battlefield  applications  were  presented  to  Paul 
Dumanoir  and  Pamela  Woodard,  program  directors  at  STRICOM  in  Orlando. 

•  Our  systems  have  been  used  in  a  variety  of  exercises  in  support  of  the  Marines  (in 
collaboration  with  NRL) — we  have  done  some  work  in  developing  full  3D  model, 
trees,  etc.  Displayed  results  on  laptop  with  GIS  positioning.  Integrated  mesoscale 
weather  simulation  (MM5). 


2.4  Results  on  Decision  Making  with  Uncertainty 

This  research  thrust  addressed  the  development  of  methodologies  for  handling 
uncertainty  in  space  and  time-conscious  battlefield  scenarios.  Analyses  and  algorithms 
were  developed  to  address  dynamically  changing  environments  to  assist  detection, 
tracking  and  visualization  tasks.  The  highlights  of  this  work  are  summarized  below. 

2.4.1  Temporal  Uncertainty  Reasoning: 

We  developed  an  approach  based  on  Bayesian  networks  for  time-sensitive  belief 
propagation  in  the  context  of  dynamically  changing  environments,  where  the  relevance  of 
observations  decays  with  time.  A  compact  and  efficient  continuous-time  representation 
is  used,  unlike  the  dominant  “time-slice”  approach  predominant  in  the  literature. 
Conditional  probabilities  associated  with  edges  in  the  networks  depend  on  time  in  two 
ways:  the  exact  timestamp  associated  with  individual  nodes  in  the  network,  and  the  time 
delays  between  observations  and  other  hypotheses  associated  with  nodes.  The  network 
propagation  model  has  been  implemented  along  with  a  visualized  battlefield  tracking 
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application  where  uncertain  observations  of  tracked  entities  are  made  using  multiple 
sensors. 

2.4.2  Decision  Fusion  in  a  Large  WSN: 

For  a  wireless  sensor  network  (WSN)  deployed  in  the  battlefield  with  a  random  number 
of  sensors,  we  have  proposed  a  decision  fusion  rule  that  uses  the  total  number  of 
detections  reported  by  local  sensors  as  a  statistic  for  hypothesis  testing.  We  assume  that 
the  signal  power  attenuates  as  a  function  of  the  distance  from  the  target,  the  number  of 
sensors  follows  a  Poisson  distribution,  and  the  locations  of  sensors  follow  a  uniform 
distribution  within  the  region  of  interest  (ROI).  Both  analytical  and  simulation  results  for 
system-level  detection  performance  have  been  provided.  This  fusion  rule  can  achieve  a 
very  good  system-level  detection  performance  even  at  very  low  signal  to  noise  ratio 
(SNR),  as  long  as  the  average  number  of  sensors  is  sufficiently  large.  For  all  the  different 
system  parameters  we  have  explored,  the  proposed  fusion  rule  is  equivalent  to  the 
optimal  fusion  rule,  which  requires  much  more  prior  information.  The  problem  of 
designing  an  optimum  local  sensor-level  threshold  has  been  investigated.  For  various 
system  parameters,  the  optimal  thresholds  have  been  found  numerically  by  maximizing 
the  deflection  coefficient.  Guidelines  on  selecting  the  optimal  local  sensor-level  threshold 
have  also  been  provided. 

2.4.3  Fusion  of  Decisions  Corrupted  by  Fading  Channels: 

We  have  investigated  the  decision  fusion  problem  for  decisions  transmitted  over  noisy 
and  fading  wireless  channels.  In  a  conventional  distributed  detection  system,  it  is  often 
assumed  that  the  decisions  sent  from  local  sensors  are  perfectly  recovered  at  the  fusion 
center.  However,  because  of  the  harsh  and  hostile  battlefield  environment,  the  power  of 
transmitted  signal  should  be  kept  to  a  minimum  to  attain  a  low  probability  of 
intercept/detection  (LPI/LPD).  As  a  result,  for  a  WSN  deployed  in  a  battlefield,  the 
transmitted  information  has  to  endure  both  channel  fading  and  noise/interference,  since 
error  protection  via  channel  coding  and  increased  transmitter  power  may  not  be  desirable 
due  to  limited  resources.  We  have  developed  a  decision  fusion  rule  that  intelligently  deals 
with  channel  fading,  and  that  provides  robust  detection  performance  in  the  presence  of 
channel  noise  and  channel  fading. 

2.4.4  Temporal  Staggering  for  Target  Tracking: 

Several  practical  battlefield  target  tracking  scenarios  require  the  periodic  collection  of 
uncertain  infonnation  from  multiple  sensors,  with  different  measurement  noise  variances. 
Decisions  need  to  be  made  regarding  the  optimal  strategies  for  collecting  data  over  time 
from  different  sensors.  We  have  studied  the  effects  of  temporally  staggered  sensors  on 
system  performance,  and  evaluated  them  by  comparison  to  synchronous  sensors. 
Optimal  staggering  patterns  have  been  found  numerically  in  some  simulations.  Practical 
guidelines  on  selecting  optimal  staggering  patterns  have  been  obtained  for  different  target 
tracking  scenarios. 
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2.4.5  Distributed  Sequential  Detection: 

We  have  considered  the  problem  of  distributed  sequential  detection  in  the  presence  of 
communication  constraints  for  a  sensor  network.  The  observations  available  at  each 
sensor  are  first  compressed  to  multi-bit  sensor  decisions  and  sent  to  the  fusion  center.  At 
the  fusion  center,  a  sequential  data  fusion  scheme  is  implemented  in  order  to  reach  a 
global  decision.  An  algorithm  has  been  developed  for  optimal  bandwidth  distribution 
among  sensors  under  a  fixed  bandwidth  constraint.  Under  symmetry  and  conditional 
independence  assumptions,  the  algorithm  can  be  simplified  substantially:  the  cooperative 
quantizer  design  algorithm  reduces  to  independent  quantizer  design.  The  case  when 
communication  bandwidth  is  the  only  constraint  has  also  been  considered. 

2.4.6  Activity  Change  Recognition: 

Visualization  in  dynamic  battlefield  environments,  using  information  from  multiple 
sensors  observing  multiple  locations,  necessitates  automatically  focusing  attention  on  the 
regions  of  interest  where  important  changes  have  occurred  in  the  activities  of  individuals 
being  monitored.  We  have  addressed  this  problem,  developing  a  new  Control  Charts 
approach  for  activity  change  recognition  using  human  activity  data  from  video  image 
sequences. 

2.4.7  Heterogeneous  Sensor  Data  Fusion: 

In  battlefield  surveillance-related  applications,  asynchronously  arriving  data  from 
heterogeneous  sensors  needs  to  be  combined  effectively  in  order  to  draw  some  inferences 
that  cannot  be  drawn  from  individual  data  streams  by  themselves.  This  problem  is 
particularly  difficult  when  the  data  streams  are  incommensurate  and  substantially 
different.  We  have  addressed  this  problem  by  developing  a  heterogeneous  sensor  fusion 
methodology  with  multiple  sensor-specific  data  processing  pipelines  that  send 
information  asynchronously  to  the  decision-maker,  which  bases  its  current  conclusions 
on  all  the  data  received  thus  far  from  various  data  streams.  We  have  applied  this 
approach  to  the  task  of  fusing  audio  and  video  data  streams,  classifying  entities  in  the 
environment  and  detecting  the  activities  of  multiple  individuals,  using  neural  networks 
and  fuzzy  rules  for  classification  decisions.  Situation-specific  domain  knowledge  such  as 
information  regarding  the  positions  of  certain  scene  landmarks  such  as  position  of  cars 
etc,  helps  to  provide  a  richer  understanding  of  the  scene  context  and  ability  to  spot 
specific  events. 

2.4.8  Distributed  Particle  Filters: 

Noisy  sensor  measurements  result  in  detection  failures  and  false  alarms  when  tracking 
numerous  indistinguishable  targets  appearing  at  random  in  space  and  time.  We  have 
developed  distributed  particle  filter  techniques  to  address  these  problems,  improving  the 
accuracy  of  non-linear/non-Gaussian  tracking  problems  in  a  multi-sensor  environment, 
where  the  (Extended)  Kalman  Filter  approach  often  fails.  To  save  the  communication 
bandwidth,  we  introduce  a  modified  Expectation  Maximization  algorithm  to  compress 
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the  local  tracking  information  before  sending  it  to  the  fusion  center.  A  corresponding 
fusion  approach  is  also  introduced  to  fuse  the  information  collected  from  each  sensor  and 
generate  the  sub-optimal  estimation  about  the  moving  targets.  We  have  also  addressed 
the  maximization  of  network  lifetime  for  target  tracking  by  intelligently  activating  the 
most  appropriate  sensors  and  deactivating  others  to  save  energy. 

2.4.9  Vehicle  Movement  Uncertainty  Visualization: 

In  collaboration  with  MURI  team-members  from  Univ.  of  California  at  Santa  Cruz,  we 
have  developed  techniques  for  the  visualization  of  uncertainty  associated  with  the 
locations  of  objects  such  as  vehicles  moving  on  road  segments.  An  important  aspect  of 
this  work  is  that  the  associated  uncertainty  changes  with  space  and  time,  generally 
depending  on  the  reliability  of  the  sensed  information  and  the  timestamps  of 
observations. 

2.4.10  Personnel  movement: 

An  important  scenario  for  the  application  of  various  uncertainty  reasoning  and 
visualization  algorithms  is  the  task  of  planning  personnel  movements  in  the  battlefield, 
with  uncertain  information  arriving  over  time  from  multiple  spatially  distributed  sensors 
or  observers.  We  have  addressed  this  task,  developing  algorithms  for  fast  computation  of 
near-optimal  paths,  updated  rapidly  when  new  infonnation  arrives  after  personnel 
movement  has  been  initiated.  We  have  developed  a  new  hierarchical  path  planning 
algorithm  (HIPLA)  for  fast  computation  of  near  optimal  paths  for  such  applications, 
which  incorporates  the  dynamic  aspect  as  an  integral  part  of  the  path  planning  model. 
When  the  risk  estimates  associated  with  nodes  change  with  time  after  personnel  have 
started  traversing  a  path,  HIPLA  computes  a  new  sub-path  from  their  current  location  to 
destination  nodes,  reusing  the  previously  computed  risk  estimates  for  the  unaffected 
portions  of  the  graph,  saving  significant  computational  time  as  opposed  to  re-computation 
of  the  whole  path  in  the  original  graph.  Simulations,  carried  out  with  graphs  containing 
upto  19,600  nodes,  showed  that  HIPLA  outperforms  two  state-of-the-art  algorithms  viz., 
Shortest  path  algorithm  (SPAH)  and  Dijkstra’s  algorithm  with  pruning  (DP),  achieving 
near-optimal  solutions  with  substantially  less  computational  effort.  For  personnel 
movement  scenarios  where  multiple  objectives  need  to  be  optimized  (e.g.,  cost,  time,  risk 
to  personnel,  and  utility  associated  with  reaching  destination),  we  have  developed  a  new 
multi-objective  evolutionary  algorithm  (EMOCA)  and  shown  that  it  outperforms  state-of- 
the-art  algorithms  such  as  NSGA-II.  We  have  collaborated  with  MURI  team-members 
(from  Georgia  Tech./Univ.  of  N.  Carolina)  to  apply  these  algorithms  to  experiments 
involving  the  actual  movements  of  individuals  (being  tracked)  in  a  University  Campus. 

2.5  Results  on  Uncertainty  Visualization 

Our  vision  is  that  all  information  and  decisions  should  include  a  measure  of  confidence. 
This  measure  of  confidence,  or  uncertainty,  may  be  encoded  directly  with  the 
visualization,  or  it  may  be  provided  to  the  user  in  another  form.  In  summary,  we  have 
addressed  the  issue  of  credibility  in  visualizations  and  decision-making  processes 
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emerging  from  a  mobile  augmented  battlespace  visualization  system.  Our  efforts  in  this 
area  include  the  following  contributions. 

2.5.1  Computation  and  Visualization  of  Uncertainty  for  Terrains  while  Preserving 
Topological  Features  within  the  GIS  context 

Here  we  include  a  brief  summary  of  our  efforts  on  computing  and  visualizing  uncertainty 
related  to  the  compression  of  large  terrain  data  sets.  We  present  an  algorithm  for 
compressing  terrain  data  that  preserves  topology.  We  use  a  decimation  algorithm  that 
simplifies  the  given  data  set  using  hierarchical  clustering.  Topology  constraints  along 
with  local  error  metrics  are  used  to  ensure  topology  preserving  compression  and  compute 
precise  error  bounds  in  the  compressed  data.  Earth's  mover  distance  is  used  as  a  global 
metric  to  compute  the  degradation  in  topology  as  the  compression  proceeds.  Results 
indicate  that  one  can  obtain  significant  compression  with  low  uncertainty  without  losing 
topology  infonnation.  During  the  first  year  of  effort,  we  focused  on  preserving  point 
features  such  as  high  points,  low  points,  or  transition  points.  During  the  second  year  of 
effort,  we  have  extended  the  results  to  line  features  such  as  isocontours  or  polyline  data 
that  may  be  relevant  in  a  battlespace  scenario.  Since  global  uncertainty  computation  for 
preserving  line  features  is  very  expensive,  we  have  designed  an  approximate  local 
computation  algorithm  that  works  well  in  practice. 

2.5.2  Uncertainty-Driven  Target  Tracking 

We  have  created  visualization  of  uncertain  information  associated  with  target  tracking. 

As  a  first  step,  we  modeled  the  uncertainty  associated  with  the  location  and  velocity  of 
targets  as  probability  distributions.  We  have  visualized  the  uncertain  location  of  the  target 
as  a  blob,  which  can  be  tracked  over  time.  We  discuss  the  algorithmic  complexity  of  the 
algorithm  for  uncertainty  computation,  and  ways  to  improve  its  performance.  Three 
visualization  techniques  (galaxy,  transparency,  and  pseudo-color)  are  developed  to 
represent  the  resulting  probability  distribution  associated  with  the  particle  at  a  later  time. 
An  appropriate  time-dependent  sampling  approach  is  adopted  to  make  the  visualizations 
more  comprehensible  to  the  human  viewer.  Experiments  with  different  distributions 
indicate  that  the  resulting  visualizations  often  take  the  form  of  recognizable  real-world 
shapes,  assisting  the  user  in  understanding  the  nature  of  a  particle's  movement.  This  work 
was  done  in  collaboration  with  the  Syracuse  University. 

We  investigated  an  on-line  expert  algorithm  to  compute  uncertainty  of  mobile  GPS- 
equipped  objects.  The  algorithm  uses  the  intelligence  gathered  by  experts  to  decipher  a 
pattern  and  predict  the  movements  of  objects.  Depending  upon  the  movement,  experts 
given  to  the  weights  are  updated  in  real-time  to  track  the  objects. 

2.5.3  Embedding  Uncertainty  within  VGIS 

This  effort  required  collection,  rectification,  and  registration  of  various  GIS  data  sets 
related  to  Santa  Cruz  region.  We  now  have  many  different  types  of  GIS  data  sets 
including  A  foot  resolution  imagery  data,  elevation  data,  detailed  AUTOCAD  drawings, 
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street  maps,  and  LIDAR  data  for  parts  of  this  region.  We  have  also  successfully  inserted 
the  imagery  and  elevation  data  within  the  VGIS  (Virtual  Geographic  Information 
System)  developed  at  the  Georgia  Institute  of  Technology.  We  have  been  able  to 
visualize  uncertainty  of  mobile  objects  within  the  VGIS  system.  This  work  is  in 
collaboration  with  the  Georgia  Institute  of  Technology. 

2.5.4.  Heterogeneous  Uncertainty  within  a  GIS  Environment 

We  have  investigated  and  visualized  spatio-temporal  GPS  uncertainty  and  uncertainty 
arising  due  to  heterogeneous  geo-spatial  data.  Importance  of  accurate  registration  of  GPS 
(Global  Positioning  System)  tracked  objects  embedded  within  a  GIS  (Geographic 
Information  Systems)  context  has  emerged  as  a  critical  need  in  several  land,  marine,  and 
air  navigational  systems  both  for  civilian  and  defense  applications.  The  objective  of  this 
work  is  to  measure,  model  and  geo-spatially  register  the  positional  accuracy  of  objects 
carrying  GPS  receivers  against  a  GIS  background.  Although  positional  accuracy  is 
affected  by  a  number  of  factors,  in  this  work  we  have  focused  on  GPS  modes 
(standalone  or  differential),  type  of  environment  (urban  or  foliage),  and  type  of  expected 
movement  of  objects.  The  Ashtech  Z-12  sensor  is  used  to  collect  the  data.  Linear  models 
are  used  to  estimate  the  errors  associated  with  the  horizontal  position  information.  This 
error  is  then  visualized  upon  a  1/2  foot  resolution  aerial  imagery  of  the  UCSC  (University 
of  California,  Santa  Cruz)  campus.  Estimates  of  speed  and  direction  errors  are  used  to 
create  visualizations  of  spatio-temporal  uncertainty  associated  with  an  object  walking 
through  the  campus. 

GIS  data  sets  acquired  using  different  sensors  at  different  times  such  as  Aerial  Lidar  data, 
ground  Lidar  data,  aerial  imagery,  ground  level  imagery,  Autocad  drawings  do  not 
register  accurately  with  respect  to  each  other.  Sources  of  error  are  numerous  including 
sensor  noise,  geo-spatial  registration  inaccuracies,  shifting  of  ground  over  time  due  to 
earthquakes,  and  changing  lighting  conditions.  We  compute  and  display  uncertainty  in 
GIS  data  with  a  view  to  create  a  common  consistent  view  for  decision  making. 

2.5.5  Uncertainty  in  Cross-Registration  of  Data  acquired  at  Different  Resolutions 

Adaptive  fusion  of  new  information  in  a  3D  urban  scene  is  an  important  goal  in 
battlespace  visualization.  In  this  work  we  acquire  new  image  pairs  of  a  scene  from  closer 
distances  and  extract  3D  models  of  successively  higher  resolutions.  We  present  a  new 
hierarchical  approach  to  register  these  texture-mapped  3D  models  with  a  coarse  3D 
texture  mapped  model  of  an  urban  scene.  First,  we  use  the  standard  reconstruction 
algorithm  to  construct  3D  models  after  establishing  1-1  correspondence  between  the 
feature  points  of  two  images  at  same  resolution.  Next,  a  subset  of  these  feature  points  is 
used  to  register  the  higher  resolution  image  with  the  lower  resolution  image  using  a  scale- 
sensitive  algorithm.  Finally  we  register  and  consistently  merge  the  3D  models  at  different 
resolutions.  We  have  applied  this  algorithm  successfully  for  adaptive  enhancement  of 
scenes  by  registering  data  that  differ  in  scale  by  a  ratio  of  1:15.  Results  indicate  that  the 
proposed  hierarchical  registration  technique  effectively  utilizes  the  intermediate  models 
to  enable  the  smooth  registration  of  the  high  resolution  models  on  the  coarser  models. 
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2.5.6  Multi-modal  Exploration  of  Uncertainty  in  Geo-Spatial  Environment 

We  have  investigated  use  of  multimodal  means  to  convey  uncertainty  in  an  already 
cluttered  and  overloaded  visualization  using  expert  systems,  sound  and  speech  using 
anticipatory  multimodal  interfaces.  The  mobile  user  is  able  to  view  the  geo-spatial 
database  interactively  and  is  able  to  communicate  using  limited  speech  vocabulary  and 
receive  spoken  feedback.  Mobile  users  are  able  to  communicate  with  each  other  using 
wireless.  We  have  also  developed  anticipatory  capabilities  to  design  agents  that  can  leam 
user  profdes  to  provide  proactive  feedback  to  the  user. 

2.5.7  A  Generic  Structure-from-Motion  Algorithm  for  Cross-Camera  Scenarios 

In  this  work,  we  first  introduce  a  generic  structure-from-motion  approach  based  on  a 
highly  generic  imaging  model,  where  cameras  are  modeled  as  possibly  unconstrained  sets 
of  sets  of  projection  rays.  This  approach  unifies  most  existing  camera  types  including 
pinhole  cameras,  sensors  with  radial  or  more  general  distortions,  and  catadioptric 
cameras  (central  or  non-central).  We  are  able  to  reconstruct  3D  scenes  from  calibrated 
images  possibly  taken  by  cameras  of  different  types.  The  question  is  how  accurate  is  this 
reconstruction? 

We  have  proposed  two  approaches  for  increasing  the  accuracy  of  reconstruction 
obtaining  optimal  solutions  using  bundle  adjustment.  In  these  approaches,  camera 
motion,  calibration  and  3D  point  coordinates  are  refined  simultaneously.  The  first 
straightforward  approach  minimizes  distances  between  3D  points  and  projection  rays. 
The  other  more  promising  approach  minimizes  re-projection  error.  However,  the  general 
imaging  model  does  not  provide  analytical  expressions  for  the  re-projection  error  and  its 
derivatives,  which  are  desirable  for  efficient  optimization.  Therefore,  we  proposed  to 
approximate  the  set  of  projection  rays  of  a  general  non-central  camera  by  several  clusters 
of  central  rays,  allowing  us  to  formulate  an  analytical  cost  function. 

We  presented  results  for  two  cross-camera  scenarios  —  a  pinhole  used  together  with  an 
omnidirectional  camera  and  a  stereo  system  used  with  an  omnidirectional  camera.  Using 
ground-truth  and  3D  reconstruction  results  from  classical  techniques,  we  show  that  our 
generic  algorithm  is  simple,  general  and  accurate  for  extensions  to  various  cross-camera 
and  multi-camera  scenarios. 

2.5.8  Ground  Classification  using  Expectation-Maximization 

In  this  work,  we  classify  3D  aerial  LiDAR  height  data  into  roads,  grass,  buildings,  and 
trees  using  a  supervised  expectation-maximization  (EM)  algorithm.  We  have  computed 
and  visualized  the  uncertainty  in  classification  using  confusion  matrices. 

We  present  an  overview  of  our  approach.  Since  the  terrain  is  highly  undulating,  we 
subtract  the  terrain  elevations  using  digital  elevation  models  (DEMs,  easily  available 
from  the  United  States  Geological  Survey  (USGS))  to  obtain  the  height  of  objects  from  a 
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flat  level.  In  addition  to  this  height  information,  we  use  height  texture  (variation  in 
height),  intensity  (amplitude  of  lidar  response),  and  multiple  (two)  returns  from  lidar  to 
classify  the  data.  Furthermore,  we  have  used  luminance  (measured  in  the  visible 
spectrum)  from  aerial  imagery  as  the  fifth  feature  for  classification.  We  have  used 
mixture  of  Gaussian  models  for  modeling  the  training  data.  Model  parameters  and  the 
posterior  probabilities  are  estimated  using  Expectation-Maximization  (EM)  algorithm. 

We  experimented  with  different  number  of  components  per  model  and  found  that  four 
components  per  model  yield  satisfactory  results.  We  have  tested  the  results  using  leave- 
one-out  as  well  as  random  $\frac{n}  {2} $  test.  Classification  results  are  in  the  range  of 
66%  —  84%  depending  upon  the  combination  of  features  used  that  compares  very 
favorably  with  train-all-test-all  results  of  85\%.  Further  improvement  is  achieved  using 
spatial  coherence. 

2.5.9  Ground  Classification  using  Support  Vector  Machines  (SVM) 

We  classify  3D  aerial  lidar  scattered  height  data  into  buildings,  trees,  roads,  and  grass 
using  the  Support  Vector  Machine  (SVM)  algorithm.  We  have  used  five  features  —  height 
data,  height  variation,  nonnal  variation,  lidar  intensity  returns,  and  image  intensity  —  to 
classify  the  data  into  four  classes.  We  also  use  only  height-derived  features  —  height, 
height  variation,  and  normal  variation  —  to  classify  the  data  into  three  categories  — 
buildings,  trees  (high  vegetation),  and  road-grass.  We  have  implemented  and 
experimented  with  several  variations  of  the  SVM  algorithm  with  soft-margin 
classification  to  allow  for  the  noise  in  the  data.  We  have  applied  our  results  to  classify 
aerial  lidar  data  collected  over  approximately  8  square  miles.  We  visualize  the 
classification  results  along  with  the  associated  confidence  using  a  variation  of  the  SVM 
algorithm  producing  probabilistic  classifications.  We  observe  that  the  results  are  stable 
and  robust.  We  compare  the  results  against  the  ground  truth  and  obtain  higher  than  90% 
accuracy  and  convincing  visual  results. 

2.5.10  Ground  Classification  using  AdaBoost 

We  use  the  AdaBoost  algorithm  to  classify  3D  aerial  lidar  scattered  height  data  into  four 
categories:  road,  grass,  buildings,  and  trees.  To  do  so  we  use  five  features:  height,  height 
variation,  normal  variation,  lidar  return  intensity,  and  image  intensity.  We  also  use  only 
lidar-derived  features  to  organize  the  data  into  three  classes  (the  road  and  grass  classes 
are  merged).  We  apply  and  test  our  results  using  ten  regions  taken  from  lidar  data 
collected  over  an  area  of  approximately  eight  square  miles,  obtaining  higher  than  92\% 
accuracy.  We  also  apply  our  classifier  to  our  entire  dataset,  and  present  visual 
classification  results  both  with  and  without  uncertainty.  We  implement  and  experiment 
with  several  variations  within  the  AdaBoost  family  of  algorithms.  We  observe  that  our 
results  are  robust  and  stable  over  all  the  various  tests  and  algorithmic  variations.  We  also 
investigate  features  and  values  that  are  most  critical  in  distinguishing  between  the  classes. 
This  insight  is  important  in  extending  the  results  from  one  geographic  region  to  another. 
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•  Best  paper  award:  R.  Niu,  P.  Varshney,  M.H.  Moore,  and  D.  Klamer,  “Decision 
Fusion  in  a  Wireless  Sensor  Network  with  a  Large  Number  of  Sensors”,  Seventh 
International  Conference  on  Information  Fusion,  Stockholm,  Sweden,  June  2004. 

•  P.  Varshney:  IEEE  Distinguished  Lecturer  for  AES  Society.  Have  lectured  at 
Rochester,  Long  Island,  Syracuse,  Waterloo  and  Atlanta  Sections. 

•  P.  Varshney:  Plenary  lectures  at  National  Systems  Conf.  in  India  and  Passive 
Covert  Radar  Conf.,  2005. 

•  A.  Zakhor:  Plenary  talk  at  the  registration  workshop  at  CVPR  2005 

•  W.  Ribarsky:  Bank  of  America  Endowed  chair  at  UNC  Charlotte 

•  June  2005,  R&D  magazine  on  the  Berkeley  3D  modeling  system: 
http://www.rdmag.com/ShowPR.aspx7PUBCODEM)  14&ACCT=1400000100&I 

SSUE=0506&RELTYPE=PR&ORIGRELTYPE=CVS&PRODCODE=OOOOOOOQ 

&PRODLETT=H 

•  May  2005  New  Scientist  on  the  Berkeley  3D  modeling  system: 
http://www.newscientist.com/article.ns?id=mgl8624985.800&feedld=online- 

news  rss20 

•  American  Public  Radio  Interview,  Future  Tense,  May  2005  on  the  Berkeley  3D 
modeling  system:  http://tinyurl.com/dctc6 
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Appendix  II:  Technology  Transitions 

•  New  Darpa  program  UrbanScape  was  initiated  based  on  this  Muri: 

o  http://dtsn. darpa.  mil/ixo/programs.asp‘?id=86 

o  Program  managers: 

■  Dr.  Tom  Stratt 

■  Dr.  Brian  Leininger 

•  Start  up  founded  by  post-doc  supported  by  this  Muri: 

o  Dr.  Christian  Frueh 
o  Urban-Scan:  http://www.urban-scan.com 

o  Urban-Scan  subcontractor  to  Darpa  UrbanScape  through  SAIC  formerly 
GSTI-3D 

•  3D  city  modeling  project  software  has  been  submitted  to  Office  of  Technology 
and  Licensing  at  Berkeley  for  licensing 

o  Urban-Scan  licensed  software  from  UCB  OTL 
o  Google  and  Computa  Labs  in  negotiations  to  license 

•  Two  day  tutorial  at  Berkeley  to  ARL  personnel  on  operational  and  algorithmic 
aspects  of  3D  city  modeling  software 

o  Provided  complete  video/audio/text  documentation  of  city  modeling 
project 

•  Carried  out  a  2  day  3D  modeling  of  Potomac  Yard  Mall  in  Washington,  DC  in 
December  2003  for  Jeff  Turner  of  then  GSTI-3D,  now  SAIC 

•  Carried  out  2  day  modeling  of  Ft.  McKenna  in  Geogia  in  December  2003  in 
collaboration  with  Jeff  Dehart  of  the  ARL 

o  Delivered  the  3D  model  to  Larry  Tokarcik’s  group  at  ARL 

•  Metrolaser  Inc 

o  Provided  3D  models  for  developing  3D  holographic  displays 
o  DARPA  SBIR  on  3D  display  visualization  under  Tom  Stratt  and  Brian 
Leininger 

•  Google  Earth  provided  further  funding  to  continue  3D  modeling  effort  resulting 
from  this  MURI  and  is  in  the  process  of  licensing  Berkeley’s  3D  city  modeling 
software 

•  Darpa/SRI  VisBuilding  project  under  the  leadership  of  Dr.  Ed  Paranowski  will 
utilize  and  extend  results  from  this  MURI. 

•  Sentinel  AVE  LLC 

o  Startup  with  Pis  from  USC 
o  AVE  and  Modeling  software  systems 

■  Licensed  from  USC 
o  In  development  with 

■  Northrop  Grumman  -  perimeter  security 

■  Chevron  -  asset  security  and  operations 

■  Raytheon  -  surveillance  and  security 

■  Olympus  -  research  lab  support 

•  ARMY  TEC 

o  Lidar  modeling  system 
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•  ANDRO  Computational  Solutions  received  several  SBIR  projects  in  the  areas  of 
image  registration  and  sensor  fusion  for  missile  tracking  and  recognition  from 
AFRL  and  Missile  Defense  Agency,  POC  Mr.  A.  Drozd 

•  Two  STTR  projects  awarded  related  to  MURI 

o  AFOSR  with  JHM  technolgies 

o  ONR  with  BAE  systems/Alpha  Tech 

•  Sensis  Corp.  is  incorporating  the  temporal  Bayesian  network  model  developed 
under  this  MURI  in  their  situational  awareness  engine,  POC  Mr.  A.  Biss. 

•  Consultation  and  evaluation  with  Chubb  (UTRC)  on  sensor  fusion  for  security 
applications 

•  Technology  transfer  to  Syracuse  Center  of  Excellence  on  Environment  and 
Energy  Systems,  sponsored  by  the  state  of  NY  on  design  of  its  fully  instrumented 
headquarters  building. 

•  Coop  Research  Agreement  with  Army  Research  Laboratory  in  the  area  of 
personnel  detection  based  on  the  work  on  heterogeneous  sensor  fusion,  POC  Dr 
T.  Damarla 

•  Air  Force  Office  of  Scientific  Research  Grant  in  the  area  of  information  fusion 
and  infonnation  exploitation,  POC  Dr  A.  Magnus 

•  Office  of  Naval  Research/Oak  Ridge  National  Laboratory  Coop  Research 
Agreement  in  the  area  of  multi-domain  networks  for  detection  of  nuclear  radiation 
and  for  perimeter  surveillance,  POC  Dr.  N.  Rao 

•  Presented  mobile  situational  visualization  and  its  relation  to  homeland  security: 

o  Special  session  of  the  AAAS  meeting  on  the  National  Visualization  and 
Analytics  Center,  February,  2005,  at  AppliedVis  2005,  May,  2005 

o  DHS  Regional  Visualization  and  Analytics  Centers,  January,  2006 

•  Spun  off  a  new  ARO  funded  project  on  eye  point  dependent  models  applied  to 
terrain  analysis: 

o  Based  on  urban  terrain  analysis  in  this  MURI. 

•  Established  the  Southeastern  Regional  Visualization  and  Analytics  Center,  funded 
by  DHS: 

o  critical  infrastructure  simulations  for  disaster  relief  planning  and 
emergency  response. 

o  Will  use  terrain  visualization  and  modeling  capabilities  developed  in 
MURI. 

•  Bank  of  America  anti -money-laundering  project 

o  Bank  activity  visualization  on  geo-political  structure 

•  Samoffi  Collaboration  on  uncertainty  visualization  and  probabilistic  moving 
targets 

•  NASA  AMES:  Collaboration  on  multimodal  situational  visualization  and 
software  development 

•  Raytheon:  Transition  software  on  low  uncertainty  compression/registration 
algorithms  for  terrain. 
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Appendix  IV:  Publications 


BOOKS  (Total:  2) 

1.  G.L.  Foresti,  C.S.  Regazzoni  and  P.K.  Varshney  (Eds.),  Multisensor  Surveillance 
System:  The  Fusion  Perspective  ,  Kluwer  Academic  Press,  2003. 

2.  “Illuminating  the  Path”,  Research  Agenda  for  visualization  and  analytics  program, 
2005,  Thomas  and  Cook. 

BOOK  CHAPTERS  (Total:  4) 

1 .  Peter  Sturm,  Srikumar  Ramalingam,  and  Suresh  K.  Lodha,  "On  Calibration,  Structure 
from  Motion  and  Multi-View  Geometry  for  Generic  Camera  Models",  in  Imaging 
Beyond  the  Pin-hole  Model,  K.  Daniilidis,  R.  Klette,  and  A.  Leonardis  (editors), 
Kluwer  Academic  Publishers,  2005. 

2.  William  Ribarsky.  Virtual  Geographic  Infonnation  Systems.  The  Visualization 
Handbook,  Charles  Hanson  and  Christopher  Johnson,  editors  (Academic  Press,  New 
York,  2003). 

3.  Development  of  Tools  for  Construction  of  Urban  Databases  and  Their  Efficient 
Visualization,”  Nickolas  Faust  and  William  Ribarsky,  Modeling  and  Visualizing  the 
Digital  Earth,  Mahdi  Abdelguerfi,  Editor  (Kluwer,  Amsterdam,  2001). 

4.  William  Ribarsky,  Nickolas  Faust,  Zachary  Wartell,  Christopher  Shaw,  and  Justin 
Jang,  “Visual  Query  of  Time-Dependent  3D  Weather  in  a  Global  Geospatial 
Environment,”  Mining  Spatio-Temporal  Information  Systems,  R.  Ladner,  K.  Shaw, 
and  Mahdi  Abdelguerfi,  Editors  (Kluwer,  Amsterdam,  2002). 

PEER-REVIEWED  JOURNALS  (Total:  35) 

1.  Secord  and  A.  Zakhor,  Tree  Detection  in  Urban  Regions  Using  Aerial  LiDAR  and 
Image  Data,  IEEE  Geoscience  and  Remote  Sensing  Letters  (GRSL),  Vol.  4,  No.  2, 
pp.  196-200,  April  2007. 

2.  C.  Frueh,  S.  Jain,  and  A.  Zakhor,  " Data  Processing  Algorithms  for  Generating 
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